Repository navigation
CI: harden build-wheel.yml for the eMudhra/emSign CA incident (13.4.x backport) - #3305
Conversation
… backport) Backport the CI hardening that main received for the eMudhra/emSign CA key compromise (CVE-2026-111943, CVE-2026-115557), so that wheels released from this branch are built with patched tooling and verified downloads: - install-sccache: pin rapidsai/sccache v0.18.0-rapids.7 and verify its sha256 instead of piping releases/latest into sudo tar (NVIDIA#3248). Also use it in test-sdist-linux.yml instead of mozilla-actions/sccache-action. - GPU driver installers: verify DRIVER_SHA256 before running them as root/admin, with a test-matrix check (NVIDIA#3249). - upgrade-ca-certificates: upgrade ca-certificates on GitHub-hosted Linux and distrust the compromised intermediates on GitHub-hosted Windows, in build-wheel.yml, coverage.yml, test-sdist-windows.yml and the Windows pre-commit job in ci.yml (NVIDIA#3270). - setup-hardened-python: install interpreters with a pinned uv and fail on unpatched ones, replacing actions/setup-python + harden-pip (NVIDIA#3275). This branch still builds and tests Python 3.10, which has no fixed release; for series older than 3.11 the action warns and installs pip from the sha256-pinned 26.2.2 wheel instead of failing. cibuildwheel now runs with the hardened host Python (`python-path`). - cibuildwheel: use build[uv] with UV_* constraints, as main does (NVIDIA#2964), so build dependencies are not fetched by the pip that cibuildwheel provides. Drop the dummy wheel test stage (sccache stats are dumped during repair) and the `before-build = "pip install delvewheel"` overrides. - fetch_ctk: verify the CTK redistrib JSON against a pinned sha256 and every component archive against the sha256 in that JSON before extracting it, failing closed on a mismatch (new `component-sha256` and `verify-metadata` subcommands in ci/tools/fetch_ctk_redistrib.py, with tests). Not backported: the runner trust probe, the "Enable Git symlinks" steps, and the lychee/pre-commit workflows, which do not exist on this branch. Refs NVIDIA#3195
|
/ok to test 3541ec5 |
|
|
@coderabbitai full review |
Action failed
|
|
/ok to test 38fc040 |
…ntainer Two follow-ups from CodeRabbit review of the eMudhra hardening backport: - fetch_ctk: add 'verified' prefix to CTK_CACHE_KEY / CTK_CACHE_FILENAME so that cache entries written before the hash-verification logic was introduced cannot hit and bypass the new verify-metadata / archive SHA-256 checks. - test-wheel-linux: upgrade ca-certificates inside the ubuntu:22.04 test container before its first HTTPS operation. The host-level upgrade-ca-certificates action does not affect the container, so the container's trust store could otherwise still accept the compromised eMudhra/emSign intermediates.
|
/ok to test 8ecac04 |
rwgk
left a comment
There was a problem hiding this comment.
I had my agent look at both PRs together, asking it specifically to double-check the merge conflict resolutions. Here is what it found:
codex GPT-6.1-Sol ultra:
One confirmed omission in #3305; #3306’s conflict resolution looks correct with high confidence.
-
[P2] #3305 needs PyYAML in its CI-tool test environment. ci-nightly.yml:45–54 creates an isolated environment and installs only pytest, but the newly backported driver-hash test imports
yaml. I reproduced collection failing withModuleNotFoundError. Addpyyaml==6.0.3, asmainalready does. -
#3305 also has a probable Windows coverage regression. coverage.yml:365–369 adopts standalone Python without the stack-reserve fix from #3342. Identical subprocess tests suffered stack overflows on
main; the existing pytest worker-thread workaround doesn’t enlarge child-process stacks. This is strongly supported by the actual failure, but I haven’t reproduced it on this branch.
For #3306, comparing the original and rebased commits shows every hardening addition preserved, alongside the base branch’s Dependabot updates. I found no conflict-resolution defects.
Otherwise, the source-to-backport comparisons check out. Full CI passed at both reviewed heads (8ecac04 and b00b54e); local tooling tests passed 79 + 11. Regular PR CI doesn’t exercise the two jobs discussed above.
…ve fix Finding 1 (ci-nightly.yml): The driver-hash test added by the eMudhra backport imports `yaml`, which is not in stdlib. Add `pyyaml==6.0.3 pathspec==0.12.1` to the pip install in the CI-tools unit test step, matching what `main` does. Without this, `ci-nightly` would fail at test collection with ModuleNotFoundError. Finding 2 (coverage.yml): Integrate NVIDIA#3342 ("Fix Windows coverage test stack overflows") onto this branch. Add a `stack-reserve` input to setup-hardened-python; both Windows coverage jobs (build wheel + test wheel) request 8 MiB, letting the action patch the base interpreter with `pefile` and verify the reserve at runtime. Switch the Windows pytest steps from `run_pytest_with_stack.py` to direct `python -m pytest`, since the patched interpreter now provides a larger stack on the main thread and in child processes. The old MSVC `editbin` step is implicitly replaced by the action. Identified in review by rwgk.
|
/ok to test 455bb8a |
rwgk
left a comment
There was a problem hiding this comment.
No new findings in 455bb8a. Both previous concerns are addressed correctly, with high confidence.
- The corrected nightly dependencies pass all 79 tests in a fresh environment.
- The Windows stack fix matches #3342 exactly and is enabled in both coverage jobs. Upstream Windows logs confirm the formerly failing subprocess tests pass with it.
Description
Refs #3195
Backports the CI hardening that
mainreceived for the eMudhra/emSign CA key compromise (CVE-2026-111943, CVE-2026-115557) to13.4.x, so that wheels released from this branch are built with patched tooling and verified downloads. Before this change the branch's cibuildwheel builds used the pip frontend with an unpatched pip, installed an unverifiedreleases/latestsccache, and downloaded CTK archives and GPU driver installers without checksums.install-sccacheaction: pinrapidsai/sccachev0.18.0-rapids.7and verify its sha256 (CI: pin and verify sccache via a shared composite action #3248), and use it intest-sdist-linux.ymlinstead ofmozilla-actions/sccache-action.DRIVER_SHA256before running the installer, with a test-matrix check (CI: verify GPU driver installer SHA-256 before execution #3249).upgrade-ca-certificatesaction: run inbuild-wheel.yml,coverage.yml,test-sdist-windows.ymland the Windows pre-commit job inci.yml(CI: add eMudhra/emSign CA stopgap for GitHub-hosted Windows runners #3270).setup-hardened-pythonaction: replacesactions/setup-pythonplusharden-pipeverywhere, including the release verification jobs (CI: install eMudhra-fixed Pythons with a pinned uv #3275). This branch still builds and tests Python 3.10, which has no fixed release, so for series older than 3.11 the action warns and installs the sha256-pinned pip 26.2.2 wheel instead of failing.build[uv]withUV_*constraints and the hardened host Python, asmaindoes (build: switch to uv as build frontend #2964). This drops the dummy wheel test stage (sccache stats are dumped during repair) and thebefore-build = "pip install delvewheel"overrides in thepyproject.tomlfiles.fetch_ctk: verify the CTK redistrib JSON against a pinned sha256 and each component archive against the sha256 in that JSON before extracting it, failing closed on a mismatch. This addscomponent-sha256andverify-metadatatoci/tools/fetch_ctk_redistrib.py, with unit tests. All CUDA versions used by this branch's CI (12.6.3, 12.9.1, 13.0.2, 13.4.2) are pinned; unpinned versions only warn. This part is not onmainyet.Not backported: the runner trust probe, the "Enable Git symlinks" steps, and the lychee and
pre-commit.ymlworkflows, which do not exist on this branch.Notes for reviewers
The commit is a squash of cherry-picks with adaptations for this branch; the conflicts were in the workflow files, which differ from
main(cuda_bindings/instead ofcuda_bindings_13/, no lychee workflows, Python 3.10 still supported).Validation:
ci/tools/testspass locally (79 tests), pre-commit including actionlint passes, and the pinned redistrib hashes were re-checked against the live JSONs. The workflow changes have not run on GitHub Actions yet, so CI on this PR is the real test, in particular the Windows cibuildwheel jobs withbuild[uv]and the Python 3.10 fallback.Checklist
Summary by CodeRabbit